Testing statistical graphics

Evaluate design choices and understand cognitive biases through the use of visual tests.

Could ask participants to:

• identify differences in graphs.
• read information off of a chart accurately.
• use data to make correct real-world decisions.
• predict the next few observations.

All of these types of tests require different levels of use and manipulation of the information presented in the chart.

Task complexity

(Carpenter and Shah 1988) identifies pattern recognition, interpretative processes, and integrative processes as strategies and processes required to complete tasks of varying degrees of complexity.

• Pattern recognition requires the viewer to encode graphic patterns.

• Interpretive processes operate on those patterns to construct meaning.

• Integrative processes then relate the meanings to the contextual scenario as inferred from labels and titles.

Research objectives

Big Idea: Are there benefits to displaying exponentially increasing data on a log scale rather than a linear scale?

Lineup experimental task

Study Participant Prompt: Which plot is most different?

Lineup study design

Lineup results

‘You Draw It’ experimental task

Study Participant Prompt: Use your mouse to fill in the trend in the yellow box region.

‘You Draw It’ study design

‘You Draw It’ prediction results

‘You Draw It’ prediction results

Simulated Data

Questioning

Estimation strategy

Estimation results

Main take-aways

• Understanding log logic is difficult.

• Accuracy greatly depends on the location of the value being estimated in relation to the magnitude.

• Strong anchoring and rounding effects.

  • Participants were resistance to estimate between grid lines on the log scale.

  • Inaccurate representation of equating spatial distance to quantitative difference.

• Inaccurate first level estimations can lead to consequences in estimations which require participants to make comparisons between two points.

• Estimates were subjective to the simulated data set.

Conclusions

Perception

• Perceptual differences result from the contextual appearance (depends on choice of scale) of the trends.

Prediction

• Clear underestimation of forecasting trends with high exponential growth rates when participants were asked to make predictions on the linear scale.

Estimation

• Log logic is difficult and that we often misinterpret and miscalculate multiplicative reasoning.
• Estimation accuracy for small magnitudes was improved by the use of the log scale, but sacrifices in accuracy on the log scale became apparent as magnitudes increased leading to advantages on the linear scale.

Overall Recommendations

• Perceptual advantages of the use of log scales due to the change in contextual appearance.

• Our understanding of log logic is flawed when translating the information into context.

• We recommend consideration of both user needs and graph specific tasks when presenting data on the log scale.

• Caution should be taken when interpretation of large magnitudes is required, but advantages may appear when it is necessary to visually identify and interpret small magnitudes on the chart.

References

Lineup GLMM

Define \(Y_{ijkl}\) to be the event that participant \(l\) correctly identifies the target plot for data set \(k\) with curvature \(j\) plotted on scale \(i\).

\[\text{logit }P(Y_{ijk}) = \eta + \delta_i + \gamma_j + \delta \gamma_{ij} + s_l + d_k\]

where

• \(\eta\) is the baseline average probability of selecting the target plot.
• \(\delta_i\) is the effect of the log/linear scale.
• \(\gamma_j\) is the effect of the curvature combination.
• \(\delta\gamma_{ij}\)is the two-way interaction effect of the scale and curvature.
• \(s_l \sim N(0,\sigma^2_\text{participant})\), random effect for participant characteristics.
• \(d_k \sim N(0,\sigma^2_{\text{data}})\), random effect for data specific characteristics.

We assume that random effects for data set and participant are independent.

Feedback data

For each participant, the final data set used for analysis contains:

• \(x_{ijklm}\), \(y_{ijklm,drawn}\), and \(\hat y_{ijklm,NLS}\)

for:

• growth rate \(i = 1,2\),
• point truncation \(j = 1,2\),
• scale \(k = 1,2\),
• participant \(l = 1,...N_{participant}\), and
• \(x_{ijklm}\) value \(m = 1, ...,4 x_{max} + 1\).

Vertical residuals between the drawn and fitted values were calculated as: + \(e_{ijklm,NLS} = y_{ijklm,drawn} - \hat y_{ijklm,NLS}\).